Providing separate video and presentation streams to a recording server

ABSTRACT

A videoconference may be initiated between a plurality of endpoints. At least one of the endpoints may be coupled to a recording server, which may be configured to record the videoconference. A configuration may be selected (e.g., automatically or manually) for performing the recording. The endpoint (e.g., acting as an MCU) may transmit information to endpoints and may transmit recording information to the recording server. The recording information may be different from the videoconference information. For example, it may be in a “streaming friendly” format, at a different bit rate, encoded differently, have different inputs, etc. The manner in which the videoconference is stored and/or recorded may be based on the selected configuration. Clients may be configured to receive and display the videoconference from the recording server and may be configured to change the provided layout to different layouts, e.g., based on user input.

PRIORITY INFORMATION

This application claims benefit of priority of Indian Patent ApplicationNo. 1004/DEL/2010 titled “Recording a Videoconference Using a StreamingServer” filed Apr. 27, 2010, whose inventors were Keith C. King, BinuKaiparambil Shanmukhadas, Ashish Goyal, and Sunil George, which ishereby incorporated by reference in its entirety as though fully andcompletely set forth herein.

FIELD OF THE INVENTION

The present invention relates generally to conferencing and, morespecifically, to recording a videoconference.

DESCRIPTION OF THE RELATED ART

Videoconferencing may be used to allow two or more participants atremote locations to communicate using both video and audio. Eachparticipant location may include a videoconferencing system forvideo/audio communication with other participants. Eachvideoconferencing system may include a camera and microphone to collectvideo and audio from a first or local participant to send to one or moreother (remote) participants. Each videoconferencing system may alsoinclude a display and speaker to reproduce video and audio received fromremote participant(s). Each videoconferencing system may also be coupledto a computer system to allow additional functionality into thevideoconference. For example, additional functionality may include dataconferencing (including displaying and/or modifying a document for bothparticipants during the conference).

Present videoconferencing systems allow for some recording ability.However, to achieve such recording, a new recording endpoint must beadded, which takes up valuable screen space for participants.Additionally, the recorded material typically includes a recording ofitself (e.g., as blank space). Also, the information received by therecorder is not suitable for streaming to other users, but ratherrequires transcoding. Accordingly, improvements in videoconferencingrecording are desired.

SUMMARY OF THE INVENTION

Various embodiments are presented of a system and method for recording avideoconference.

A videoconference may be initiated between a plurality of endpoints. Atleast one of the endpoints may be coupled to a recording server, whichmay be configured to record the videoconference.

A request may be received to record the videoconference. For example, auser may provide a single user interaction to initiate or perform therecording of the videoconference. In one embodiment, the user may simplyselect a record button, e.g., on a remote, and the recording may beinitiated. Alternatively, the recording of the videoconference may beinitiated automatically, or may involve more than a single userinteraction, as desired. In some embodiments, the request may bereceived by the recording server, e.g., as a message from an MCU of thevideoconference or an endpoint of the videoconference.

One or more recording configurations may be selected (e.g.,automatically or manually) for performing the recording. The manner inwhich the videoconference is recorded (e.g., and how it is stored andpublished by the recording server) may be based on the selectedrecording configuration(s). Additionally, the recording configurationsmay determine the manner in which videoconference information isprovided to the recording server.

The endpoint (e.g., acting as an MCU) may transmit information toendpoints and may transmit information to the recording server forrecording. Alternatively, a different endpoint or videoconference devicemay be determined that has a high quality connection to the recordingserver. For example, the endpoint (or other type of device) that has thebest connection to the recording server may be used to provide therecording information to the recording server (e.g., in order to achievea high quality recording of the videoconference).

The recording information transmitted to the recording server may bedifferent from the information transmitted to the other endpoints. Forexample, the recording information may be in a “streaming friendly”format, at a different bit rate, encoded differently, have differentinputs or video content, etc. The format of the recording informationmay be affected by the selected recording configurations. For example,the format of the recording information may correspond to a specifiedformat of one of the recording configurations.

The recording information may include participant videoconferenceinformation and auxiliary videoconference information (e.g.,corresponding to a presentation presented in the videoconference). Theparticipant videoconference information and the auxiliaryvideoconference information may be provided together or separately(e.g., as separate streams) to the recording server, as desired.

In some embodiments, a timeline may be provided or generated whichspecifies when the auxiliary videoconference information was present inthe videoconference (e.g., when a presentation is only given for aportion of the videoconference). The recording information may specifyboundary information of a composite image provided to the recordingserver. For example, the boundary information may specify the boundaryof each portion in the composite image.

In some embodiments, the transcoding or recording work may bedistributed among a plurality of recording resources. For example, asecond endpoint may provide second recording information to therecording server that has a different format than the recordinginformation described above. In one embodiment, the second endpoint mayuse videoconference information of the videoconference to generate thesecond recording information, or may transcode the recording informationdescribed above, as desired. In various embodiments, other devices mayperform recording or transcoding work, e.g., when the recording serverlacks the capacity to record the videoconferences in all of the desiredformats (e.g., as specified by the recording configurations selectedabove). Thus, recording work may be distributed among a plurality ofdevices, e.g., including endpoints participating in the videoconference,as desired.

Clients of the recording server may be configured to receive and displaythe recorded videoconference. For example, the client may be configuredto display the participant videoconference information and the auxiliaryvideoconference information using a timeline provided by the recordingserver. Additionally, or alternatively, the client may be configured tochange the provided layout of the recording to different layouts, e.g.,based on user input. For example, boundary information may be providedfrom the recording server to the client to allow the client to changethe layout of the playback of the recording of the videoconference.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention may be obtained when thefollowing detailed description is considered in conjunction with thefollowing drawings, in which:

FIGS. 1 and 2 illustrate exemplary videoconferencing system participantlocations, according to an embodiment;

FIGS. 3A and 3B illustrate exemplary conferencing systems coupled indifferent configurations, according to some embodiments; and

FIG. 4 is a block diagram illustrating endpoints in a videoconference, arecording server, and clients, according to one embodiment;

FIGS. 5A and 5B illustrate exemplary screen layouts for avideoconference, according to one embodiment;

FIG. 6 is a flowchart diagram illustrating an embodiment of a method forrecording a videoconference;

FIGS. 7A-8B are diagrams illustrating flow of data between endpoints,recording servers, and clients, according to some embodiments;

FIG. 9 is a screen shot illustrating an interface for viewing a recordedvideoconference, according to one embodiment;

FIG. 10 is a flowchart diagram illustrating an embodiment of a methodfor recording a videoconference using configurations;

FIG. 11 is a screen shot of an interface for editing and creatingconfigurations, according to one embodiment;

FIG. 12 is a flowchart diagram illustrating an embodiment of a methodfor recording a videoconference in response to a single userinteraction;

FIG. 13 is a flowchart diagram illustrating an embodiment of a methodfor recording a videoconference with a separate presentation stream;

FIG. 14 is a flowchart diagram illustrating an embodiment of a methodfor recording a videoconference including boundary information;

FIG. 15 is a flowchart diagram illustrating an embodiment of a methodfor recording a videoconference based on connection quality; and

FIG. 16 is a flowchart diagram illustrating an embodiment of a methodfor recording a videoconference according to a plurality of formats.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that the drawings and detaileddescription thereto are not intended to limit the invention to theparticular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the present invention as defined by the appendedclaims. Note the headings are for organizational purposes only and arenot meant to be used to limit or interpret the description or claims.Furthermore, note that the word “may” is used throughout thisapplication in a permissive sense (i.e., having the potential to, beingable to), not a mandatory sense (i.e., must). The term “include”, andderivations thereof, mean “including, but not limited to”. The term“coupled” means “directly or indirectly connected”.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Incorporation by Reference

U.S. patent application titled “Video Conferencing System Transcoder”,Ser. No. 11/252,238, which was filed Oct. 17, 2005, whose inventors areMichael L. Kenoyer and Michael V. Jenkins, is hereby incorporated byreference in its entirety as though fully and completely set forthherein.

U.S. patent application titled “Virtual Decoders”, Ser. No. 12/142,263,which was filed Jun. 19, 2008, whose inventors are Keith C. King andWayne E. Mock, is hereby incorporated by reference in its entirety asthough fully and completely set forth herein.

U.S. patent application titled “Video Conferencing System which AllowsEndpoints to Perform Continuous Presence Layout Selection”, Ser. No.12/142,302, whose inventors are Keith C. King and Wayne E. Mock, ishereby incorporated by reference in its entirety as though fully andcompletely set forth herein.

U.S. patent application titled “Video Conferencing Device which PerformsMulti-way Conferencing”, Ser. No. 12/142,340, whose inventors are KeithC. King and Wayne E. Mock, is hereby incorporated by reference in itsentirety as though fully and completely set forth herein.

U.S. patent application titled “Video Decoder which Processes MultipleVideo Streams”, Ser. No. 12/142,377, whose inventors are Keith C. Kingand Wayne E. Mock, is hereby incorporated by reference in its entiretyas though fully and completely set forth herein.

U.S. patent application titled “Virtual Multiway Scaler Compensation”,Ser. No. 12/171,358, whose inventors are Keith C. King and Wayne E.Mock, is hereby incorporated by reference in its entirety as thoughfully and completely set forth herein.

U.S. patent application titled “Virtual Distributed Multipoint ControlUnit”, Ser. No. 12/712,947, whose inventors are Keith C. King, AshishGoyal, and Hrishikesh Gopal Kulkarni, is hereby incorporated byreference in its entirety as though fully and completely set forthherein.

Terms

The following is a glossary of terms used in the present application:

Memory Medium—Any of various types of memory devices or storage devices.The term “memory medium” is intended to include an installation medium,e.g., a CD-ROM, floppy disks, or tape device; a computer system memoryor random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, RambusRAM, etc.; or a non-volatile memory such as a magnetic media, e.g., ahard drive, or optical storage. The memory medium may comprise othertypes of memory as well, or combinations thereof. In addition, thememory medium may be located in a first computer in which the programsare executed, or may be located in a second different computer whichconnects to the first computer over a network, such as the Internet. Inthe latter instance, the second computer may provide programinstructions to the first computer for execution. The term “memorymedium” may include two or more memory mediums which may reside indifferent locations, e.g., in different computers that are connectedover a network.

Carrier Medium—a memory medium as described above, as well as a physicaltransmission medium, such as a bus, network, and/or other physicaltransmission medium that conveys signals such as electrical,electromagnetic, or digital signals.

Computer System—any of various types of computing or processing systems,including a personal computer system (PC), mainframe computer system,workstation, network appliance, Internet appliance, personal digitalassistant (PDA), smart phone, television system, grid computing system,or other device or combinations of devices. In general, the term“computer system” can be broadly defined to encompass any device (orcombination of devices) having at least one processor that executesinstructions from a memory medium.

FIGS. 1 and 2—Exemplary Participant Locations

FIG. 1 illustrates an exemplary embodiment of a videoconferencingparticipant location, also referred to as a videoconferencing endpointor videoconferencing system (or videoconferencing unit). Thevideoconferencing system 103 may have a system codec 109 to manage botha speakerphone 105/107 and videoconferencing hardware, e.g., camera 104,display 101, speakers 171, 173, 175, etc. The speakerphones 105/107 andother videoconferencing system components may be coupled to the codec109 and may receive audio and/or video signals from the system codec109.

In some embodiments, the participant location may include camera 104(e.g., an HD camera) for acquiring images (e.g., of participant 114) ofthe participant location. Other cameras are also contemplated. Theparticipant location may also include display 101 (e.g., an HDTVdisplay). Images acquired by the camera 104 may be displayed locally onthe display 101 and/or may be encoded and transmitted to otherparticipant locations in the videoconference. In some embodiments,images acquired by the camera 104 may be encoded and transmitted to amultipoint control unit (MCU), which then provides the encoded stream toother participant locations (or videoconferencing endpoints)

The participant location may further include one or more input devices,such as the computer keyboard 140. In some embodiments, the one or moreinput devices may be used for the videoconferencing system 103 and/ormay be used for one or more other computer systems at the participantlocation, as desired.

The participant location may also include a sound system 161. The soundsystem 161 may include multiple speakers including left speakers 171,center speaker 173, and right speakers 175. Other numbers of speakersand other speaker configurations may also be used. The videoconferencingsystem 103 may also use one or more speakerphones 105/107 which may bedaisy chained together.

In some embodiments, the videoconferencing system components (e.g., thecamera 104, display 101, sound system 161, and speakerphones 105/107)may be coupled to a system codec 109. The system codec 109 may be placedon a desk or on the floor. Other placements are also contemplated. Thesystem codec 109 may receive audio and/or video data from a network,such as a LAN (local area network) or the Internet. The system codec 109may send the audio to the speakerphone 105/107 and/or sound system 161and the video to the display 101. The received video may be HD videothat is displayed on the HD display. The system codec 109 may alsoreceive video data from the camera 104 and audio data from thespeakerphones 105/107 and transmit the video and/or audio data over thenetwork to another conferencing system, or to an MCU for provision toother conferencing systems. The conferencing system may be controlled bya participant or user through the user input components (e.g., buttons)on the speakerphones 105/107 and/or input devices such as the keyboard140 and/or the remote control 150. Other system interfaces may also beused.

In various embodiments, the codec 109 may implement a real timetransmission protocol. In some embodiments, the codec 109 (which may beshort for “compressor/decompressor” or “coder/decoder”) may comprise anysystem and/or method for encoding and/or decoding (e.g., compressing anddecompressing) data (e.g., audio and/or video data). For example,communication applications may use codecs for encoding video and audiofor transmission across networks, including compression andpacketization. Codecs may also be used to convert an analog signal to adigital signal for transmitting over various digital networks (e.g.,network, PSTN, the Internet, etc.) and to convert a received digitalsignal to an analog signal. In various embodiments, codecs may beimplemented in software, hardware, or a combination of both. Some codecsfor computer video and/or audio may utilize MPEG, Indeo™, and Cinepak™,among others.

In some embodiments, the videoconferencing system 103 may be designed tooperate with normal display or high definition (HD) displaycapabilities. The videoconferencing system 103 may operate with networkinfrastructures that support T1 capabilities or less, e.g., 1.5mega-bits per second or less in one embodiment, and 2 mega-bits persecond in other embodiments.

Note that the videoconferencing system(s) described herein may bededicated videoconferencing systems (i.e., whose purpose is to providevideoconferencing) or general purpose computers (e.g., IBM-compatiblePC, Mac, etc.) executing videoconferencing software (e.g., a generalpurpose computer for using user applications, one of which performsvideoconferencing). A dedicated videoconferencing system may be designedspecifically for videoconferencing, and is not used as a general purposecomputing platform; for example, the dedicated videoconferencing systemmay execute an operating system which may be typically streamlined (or“locked down”) to run one or more applications to providevideoconferencing, e.g., for a conference room of a company. In otherembodiments, the videoconferencing system may be a general use computer(e.g., a typical computer system which may be used by the general publicor a high end computer system used by corporations) which can execute aplurality of third party applications, one of which providesvideoconferencing capabilities. Videoconferencing systems may be complex(such as the videoconferencing system shown in FIG. 1) or simple (e.g.,a user computer system 200 with a video camera, input devices,microphone and/or speakers such as the videoconferencing system of FIG.2). Thus, references to videoconferencing systems, endpoints, etc.herein may refer to general computer systems which executevideoconferencing applications or dedicated videoconferencing systems.Note further that references to the videoconferencing systems performingactions may refer to the videoconferencing application(s) executed bythe videoconferencing systems performing the actions (i.e., beingexecuted to perform the actions).

The videoconferencing system 103 may execute various videoconferencingapplication software that presents a graphical user interface (GUI) onthe display 101. The GUI may be used to present an address book, contactlist, list of previous callees (call list) and/or other informationindicating other videoconferencing systems that the user may desire tocall to conduct a videoconference. The GUI may also present options forrecording a current videoconference, and may also present options forviewing a previously recorded videoconference.

Note that the videoconferencing system shown in FIGS. 1 and 2 may bemodified to be an audioconferencing system. For example, theaudioconference could be performed over a network, e.g., the Internet,using VOIP. The audioconferencing system, for example, may simplyinclude speakerphones 105/107, although additional components may alsobe present. Additionally, note that any reference to a “conferencingsystem” or “conferencing systems” may refer to videoconferencing systemsor audioconferencing systems (e.g., teleconferencing systems). In theembodiments described below, the conference is described as avideoconference, but note that the methods may be modified forutilization in an audioconference.

FIGS. 3A and 3B—Coupled Conferencing systems

FIGS. 3A and 3B illustrate different configurations of conferencingsystems. The conferencing systems may be operable to perform the methodsdescribed herein. As shown in FIG. 3A, conferencing systems (CUs) 320A-D(e.g., videoconferencing systems 103 described above) may be connectedvia network 350 (e.g., a wide area network such as the Internet) and CU320C and 320D may be coupled over a local area network (LAN) 375. Thenetworks may be any type of network (e.g., wired or wireless) asdesired.

FIG. 3B illustrates a relationship view of conferencing systems310A-310M. As shown, conferencing system 310A may be aware of CU310B-310D, each of which may be aware of further CU's (310E-310G,310H-310J, and 310K-310M respectively). CU 310A may be operable toperform the methods described herein. In a similar manner, each of theother CUs shown in FIG. 3B, such as CU 310H, may be able to perform themethods described herein, as described in more detail below. Similarremarks apply to CUs 320A-D in FIG. 3A.

FIG. 4—Videoconferencing System Block Diagram Including Recording Server

FIG. 4 is a block diagram of a videoconferencing system that includes arecording server. As shown, there may be a plurality ofvideoconferencing endpoints in a videoconference. The videoconferencemay be performed in the manner described in U.S. patent application Ser.No. 11/252,238, incorporated by reference above, although otherconfigurations are envisioned. As shown in FIG. 4, videoconferencingendpoint A may act as a host or MCU of the videoconference forvideoconferencing endpoints B and C. The videoconferencing endpoints A,B, and C may be similar to those described above with respect to FIGS. 1and 2, although other variations are envisioned. As shown,videoconferencing endpoint A may communicate with both endpoints B and Cin a bidirectional manner. For example, endpoint B may provide audio andvideo generated at the location of B and endpoint C may provide audioand video generated at the location of C. Accordingly, endpoint A mayprovide videoconferencing information back to B and C, respectively, forpresentation to participants at those locations. For example, A mayprovide a composite image of itself and C to B and a composite image ofitself and B to C. However, A may simply provide a composite image of A,B, and C, and each endpoint may present whatever layout they desire(e.g., based on user input), similar to embodiments described below.

In some embodiments, presentation (e.g., Powerpoint®, digital documents,web pages, videos, projections (e.g., of physical documents), scanneddocuments, etc.) video and/or audio may also be provided. For example, Cmay provide presentation information to A as well as video from a videoinput at location C. In some embodiments, the presentation informationmay be provided separately from the video of the location (e.g., over adifferent channel or stream), although the video could be combined,e.g., in a composite image. In turn, A may provide the video and/orpresentation information, and possibly its own video, to B.

Note that endpoint A, when providing composite images, may also provideinformation indicating how the composite image is formed, such that theother endpoints can divide and reform new composite images or screenlayouts based on the received composite image. For example, A mayindicate that the transmitted video from A to B includes four quadrants,a quadrant for A, a quadrant for C's location video, a quadrant for C'spresentation, and another quadrant, which could include any number ofvideo sources, e.g., auxiliary video (from a computer, DVD player,etc.), video from B, or from another video camera, e.g., at location A.In the embodiment shown in FIG. 5A, each endpoint also receives and(possibly) displays its own video, although other embodiments areenvisioned where this is not the case. Note that the composite imagessupplied to each endpoint may be the same or different, as desired.

By providing an indication of the layout and information provided ineach window of the layout, each receiving endpoint may be configured tochange the layout of the composite image, e.g., based on user input. Thelayout change may be performed “live” or “on demand”, e.g., in responseto a user requesting the layout change. Additionally, the layoutchanging may be performed as described in various ones of the patentapplications incorporated by reference above. Thus, the user may be ableto only display a single video source, or emphasize one of the sourcesover other ones (e.g., where the active speaker is shown larger than theother video sources). For example, as shown in FIG. 5B, the participantsmay be able to change the layout of the display based on the compositeimages provided in FIG. 5A without requiring the MCU to provide newimages. More specifically, in FIG. 5B, the participant at A has chosento view the presentation in a larger screen, with A, B, and C video insmaller screens; the participant at B has chosen to view C in the mainscreen and just A and the presentation in smaller screens; and theparticipant at C has chosen just to see A and B with equal shares of thescreen. Thus, each endpoint may be configured to present differentlayouts using the layout information sent by the host, e.g., withoutrequiring the host to send new or different composite images.

Note that while various embodiments described above refer to layoutchanges in response to user input, the layout changes may be performedautomatically, without user input requesting the layout change. Forexample, the videoconferencing endpoint may automatically change thevideoconference layout at a particular location in response to theaddition of a presentation, the removal of a presentation, the additionof a new participant, the removal of a new participant, the amount oftalking that is performed by each participant, etc. Thus, thevideoconference layout may be modified automatically or in response touser input, as desired.

Thus, as shown in FIG. 4, a videoconference may be performed betweenendpoints A, B, and C. As also shown, one of the endpoints of thevideoconference (e.g., the host or MCU A) may be coupled to a recordingor streaming server, e.g., via a network, such as the Internet, althoughin some embodiments, the recording server may be on a network that islocal to the endpoint. In another embodiment, the recording server maybe implemented as part of the MCU or one of the participating endpoints,e.g., the MCU may be implemented as a multi-processor system whereby oneor more processors are used for MCU functionality and one or moreprocessors are used for implementing the recording server. The recordingserver may be any type of computer server, as desired. In someembodiments, the MCU may provide video and audio of the videoconferenceto the recording server for recording.

Note that the audio and video provided to the recording server may beindependent from any other streams being sent to the other endpoints.For example, the audio and video provided to the recording server mayinclude every endpoint's audio and video (either combined or separately,as desired), whereas videoconference streams sent to each endpointduring the actual videoconference may not typically include its ownvideo (e.g., B may not receive its own video). Thus, in some embodimentsthe recording server may receive video and audio from of all of theparticipants, even though during the videoconference various ones (orall) of the participants may receive video and audio from only a subsetof the participants, although in other embodiments, this may not be thecase. However, in other embodiments, the videoconference informationsent for recording may exclude one or more video sources, e.g., inresponse to user input excluding them (or inversely, not including them)from being recorded.

Additionally, the videoconference information sent for recording may beat a different bit rate, encoding format, resolution, frame rate, or anyother variable than the information provided to the othervideoconferencing endpoints, as desired. For example, in someembodiments, the videoconference information sent from the MCU to therecording server may already be streaming friendly. For example, the MCUmay provide the videoconference information to the recording server in aformat that includes I frames (intra frames) or other information thatallows clients to self correct the video (e.g., in the event of lostpackets or an interruption). In one embodiment, the videoconferenceinformation may be provided in a container format that is used bystreaming players (flv, swf, wmv, mov, mp4, etc) and/or may be encodedusing the audio/video codecs that is supported by streaming players.Thus, in some embodiments, the recording server may not be required totranscode the videoconference before streaming it to clients. However,in further embodiments, the recording server may be able to transcodethe received videoconference into a plurality of formats, e.g., in orderto provide the videoconference to various types of devices, e.g., whichcannot support the original format. Alternatively or additionally, theMCU may provide a plurality of differently coded recordingscorresponding to the videoconference, e.g., to support heterogeneousclients. Thus, the information sent to the recording server may beindependent or unique from any of the other videoconferencinginformation sent from the MCU to the other endpoints during thevideoconference. Additionally, unlike some prior art solutions, wherethe recording server joins the videoconference as a participant, theinformation sent to the recording server may not include audio/videocorresponding to the recording server (e.g., where a portion of thevideoconference information sent to the recording server includes ablank portion corresponding to the recording server as a participant).Said another way, the recording server may not use a port of the MCU andthus may not be considered a “participant” to the videoconference.Accordingly, the recording server may not receive any video or compositesection that corresponds to the recording server.

As also shown, the recording or streaming server may be coupled to oneor more clients. In FIG. 4, the recording server is coupled to client 1,client 2, and client 3. The clients may be any type of client asdesired. For example, the client may be a typical computer system suchas a desktop or laptop computer system, a netbook, a tablet, a mobiledevice, such as a phone, etc. Generally, a client may be any type ofdevice that is capable of displaying video files or streams to a user.According to embodiments, described herein, the recording server maystream the videoconference to the clients, e.g., during thevideoconference or after, as desired. Similar to embodiments describedabove, the streams provided to the clients may include multiple videostreams (e.g., for each participant, presentation, etc.) and at least aportion of the video streams may be packaged in composite video(composite video here refers to a video stream that itself containsvideo from two or more endpoints and hence is a composite of the videofrom those endpoints, where the composite video is comprises a pluralityof composite images). Also similar to above, the recording server mayprovide the video streams to clients with information indicating thelayout of the composite video, thereby allowing the client to change thelayout of the displayed videoconference during playback.

Note that while the recording server is shown as coupled to onlyendpoint A, the recording server may be coupled to all or any subset ofthe endpoints of the videoconference, as desired. In some embodiments,it may be especially desirable that the recording server is coupled toand receives videoconference information from an endpoint with a highquality connection to the recording server, e.g., over a local areanetwork instead of the Internet or having the highest quality Internetconnection between the endpoint and the recording server.

Further details regarding the videoconference and recording server areprovided below.

FIG. 6—Recording a Videoconference Using a Recording Server

FIG. 6 illustrates a method for recording a videoconference using arecording server. The method shown in FIG. 6 may be used in conjunctionwith any of the computer systems or devices shown in the above Figures,among other devices. In various embodiments, some of the method elementsshown may be performed concurrently, performed in a different order thanshown, or omitted. Additional method elements may also be performed asdesired. As shown, this method may operate as follows.

In 602, a videoconference may be initiated or performed between aplurality of participants at respective participant locations. Morespecifically, the conference may be initiated between a firstparticipant using a first endpoint (e.g., at a first participantlocation) and a plurality of other participants using other endpoints(e.g., at other participant locations). Thus, endpoints may be similarto those described above regarding FIGS. 1-4, although other variationsare envisioned. The videoconference may be established according to anyof a variety of methods, e.g., the one described in patent applicationSer. No. 11/252,238, which was incorporated by reference above. Thevideoconference may utilize an instant messaging service orvideoconferencing service over the Internet, as desired.

In some embodiments, a multipoint control unit (MCU) may perform thefirst videoconference between the plurality of videoconferenceendpoints. For example, one of the endpoints (e.g., the first endpoint)may act as the MCU and may perform decoding and encoding operations onvideo information transmitted in the first videoconference between theplurality of videoconferencing endpoints. Alternatively, the MCU may bea separate videoconference device that performs the videoconference,e.g., it may not be incorporated in an endpoint of the videoconference.However, it should be noted that endpoints other than the MCU are stillconsidered as “performing” the videoconference.

In 604, the MCU may transmit videoconference information (e.g.,including video and audio) to the videoconferencing endpoints during thevideoconference, for presentation to corresponding participants. Thus,in 604 the MCU may receive videoconference information from variousendpoints and also transmit videoconference information to the variousendpoints.

In 606, an endpoint of the videoconference (e.g., the MCU) may transmitvideoconference information for recording the videoconference to arecording or streaming server during the videoconference, e.g.,concurrently with the transmission of the videoconference informationsent to the videoconferencing endpoints in 604. The endpoint may begintransmitting the videoconference information for recording in responseto user input, e.g., before or during the videoconference. For example,one of the participants in the videoconference may request that all or aportion of the videoconference be recorded. In one embodiment, theparticipant may be able to simply select a “record” button, e.g., on aremote, and may not have to provide any further input, e.g., usingpreset configurations. Further embodiments regarding recordingconfigurations and pins are provided below with respect to FIG. 10.

In one embodiment, participants in a videoconference may be able toselect an option which prevents their respective video/audio stream frombeing recorded. When this option is selected by a respectiveparticipant, that participant's audio/video is not transmitted to therecording server for recording. Alternatively, or additionally, one ofthe participants (e.g., a controlling user) may be able to select whichvideo streams of the videoconference should be recorded, e.g., using aGUI of the participant's respective endpoints, or via other means, asdesired. In further embodiments, the videoconference streams that arerecorded may be selected automatically without receiving user input toselect the streams to be recorded. For example, the selection may bemade by the recording server and/or an endpoint, e.g., recording onlythe most active talkers, the most important talkers, etc.

As indicated above, the videoconference information sent to therecording server may be independent or unique from the videoconferenceinformation sent to the videoconferencing endpoints. More particularly,the videoconference information sent to the recording server may be“streaming friendly”, i.e., the videoconference information may beprovided in a format that can be streamed from the recording server toclients with little or no modification. For example, the videoconferenceinformation sent to the recording server may include periodic I frames(iframes), as opposed to the normal videoconference streams, which mayinclude only an initial iframe, iframes based on a request from anendpoint (e.g., if the endpoint needs to recover video), and/or noiframes. Normal videoconference streams may not require these iframessince each endpoint may be configured to request a new iframe asnecessary, which may not be supported by the recording/streaming server.However, embodiments where one or more of the endpoints also includeperiodic iframes are also envisioned (although they may not be at thesame rate or periodicity as those provided to the recording server).

By including the periodic iframes, a client of the recording server mayrecover from lost packets or other transmission issues without having torerequest information from the recording server. Additionally, theclient may be able to seek different portions of the videoconference,e.g., based on user input selecting, for example, a position before orafter the current position of playback of the videoconference. In otherwords, the recording server may provide the videoconference in astreaming format, e.g., which is different from that provided to thevideoconferencing endpoints in the videoconference. Additionally, oralternatively, the videoconference information sent to the recordingserver may be sent in a streaming format (e.g., H. 264, Flash™, wmv,etc.) or may otherwise be wrapped in a streaming container. In someembodiments, the MCU may automatically download or select advancedproperties like bitrate, resolution, codecs, container formats from thestreaming/recording server and use that for generating thevideoconferencing information to be sent to the recording server.

The videoconference information transmitted to the recording server maybe sent in any of a plurality of ways. For example, each video and/oraudio stream (e.g., from each participant) may be transmitted separatelyto the recording server by the endpoint (e.g., the MCU). Alternatively,one or more video and/or audio streams may be composite, e.g., may beprovided as a composite video images to the recording server. Whencomposited images are sent to the recording server, the endpoint mayindicate the boundaries of the composite video image so that the layoutmay be changed at a later point, e.g., by streaming clients. As usedherein, a “composite video stream” may refer to a video stream thatcomprises a plurality of images that are a composite of multiple images,which when played sequentially form a video stream of the compositeimages. The boundary information of the composite image or compositevideo stream may indicate the borders of individual images within eachcomposite image.

In some embodiments, any combination of the separate and compositetransmission may be used. For example, presentation information may besent separately from the participant video, which may be a composite ofimages of all the participants. Alternatively, or additionally, therecording server may receive and/or store main and auxiliary videostreams separately. The audio information may be separated andassociated with a particular video stream (or portion of a compositeimage) or may simply be mixed together as a single audio for therecorded videoconference. In one embodiment, in a point to point callsetup, the information sent to the recording server may comprise onlythe local video source and the far end video source and correspondingaudio.

In some embodiments, while receiving or storing content which iscomposed of multiple video streams composited together (e.g., by acompositor), the recording server may request or simply receivecomposition information from the compositor and store it with therecorded stream. As indicated above, the composition information maydefine the boundaries of the source images in the composite image. Thecomposition information may change with time as new participants jointhe call, existing participants leave, presentations are added,auxiliary inputs are added, etc. In one embodiment, the compositioninformation may be stored against the timeline of the video. In furtherembodiments, the timeline may be used for indicating when secondarystreams of information are present during the videoconference. Forexample, where a presentation is provided for a period of time duringthe videoconference (but not all of the videoconference) the timelinemay indicate when the presentation occurred. This may be particularlyuseful when the presentation information is stored separately (and/or isprovided separately) to the recording server. Accordingly, the recordingserver will know when to provide the presentation information in 608below, e.g., by using the timeline.

Thus, in 606 the recording server receives the stream(s) from avideoconference device (e.g., an endpoint or MCU) and may record (orstore) the stream(s) in the memory of the recording server.

In 608, the recording server may provide the recorded videoconferenceinformation to one or more clients, such as shown in FIG. 4. Accordingto various embodiments, the recording server may provide thevideoconference information concurrent to (at the same time as) thevideoconference (e.g., broadcast live or with some delay). Additionally,or alternatively, the recording server may provide the videoconferenceinformation after the videoconference is recorded or complete. Forexample, the recording server may store a repository of videoconferenceswhich may be viewed by clients, e.g., via the Internet, e.g., using aweb site. In some embodiments, the clients may view the videoconferenceat any point during or after the videoconference, according to theirdesire.

When transmitting to clients, the recording server can send thecomposition or boundary information (e.g., with a timeline) to theclient. Note that the boundary information may be included in the videoinformation (e.g., as metadata) or separately, as desired. Accordingly,the client's player may be configured to use the boundary information tomodify the composite video stream in a variety of ways. For example, theclient may be configured to allow the user to zoom in to a specificsection (specific participant) of the videoconference. The client mayalso be allowed to move the video sections around and display them inlayouts that differ from the original composited video (e.g., similar tothe modifications shown in FIG. 5B from FIG. 5A, among otherpossibilities). In one embodiment, this may be achieved using multiplevideo players with each player zooming in to a separate section of thevideo. The players may then be scaled and if required overlaid on toachieve layouts like picture in picture.

FIGS. 7A-7C illustrate diagrams showing the transmission of informationfrom an MCU or endpoint to the recording server to a client (althoughthere may be a plurality of clients). In the embodiment of FIG. 7A, theendpoint transmits separate video streams to the recording or streamingserver. In this example, the video streams include a participant videostream and a presentation video stream. Note that the embodiment shownmay apply to a videoconference (including a plurality of participants)or may also apply to a single participant presenting a presentation,e.g., for broadcast via the recording server. As shown in FIG. 7A, thestreaming server may send the two video streams separately to theclient, which may in turn display the two video streams in any ofvarious manners, e.g., according to the user's desire. For example, asshown, the user may view both streams equally or the presentation in alarger format. Further layouts are envisioned, e.g., where only one orthe other video is displayed.

FIG. 7B illustrates a diagram showing transmission of information froman MCU or endpoint to the recording server to a client. In thisembodiment, the MCU receives videoconference information from otherendpoints and composites the video with multiple participant images. TheMCU sends this composite video as well as boundary informationidentifying the composite sections to the recording server. Therecording server in turn provides this information to the client, whichis able to view the videoconference in the original layout, or any ofvarious other layouts (e.g., displaying only a single participant oremphasizing one of the participants by showing that participant as alarger portion of the display, although other layouts are envisioned).In the embodiment of FIG. 7B, the endpoint and/or recording server maybe configured to transmit auxiliary streams as well (e.g., video streamscorresponding to a presentation), either in the composite video streamor separately, as desired.

FIG. 7C illustrates the transmission of two streams from the MCU orendpoint to another endpoint and the transmission of two streams fromthe MCU to the recording server. Thus, in the embodiment of FIG. 7C, theMCU is able to send a plurality of different streams to endpoints aswell as to the recording server. Note that, in this embodiment, therecording server streams include periodic iframes. Thus, FIGS. 7A-7Cillustrate the transmission of data to the client via the recordingserver, allowing users to change the layout of the recordedvideoconference.

FIGS. 8A and 8B illustrates two comparative embodiments showing thetransmission of data. In FIG. 8A, a recording method is shown where therecorder is a participant in the videoconference. In this embodiment,the videoconference information provided to the recording server is notunique or different from the other two videoconferencing endpoints.Additionally, the screen layout includes the recorder, which takes up aportion of the screen for the participants (and is also recorded). Therecording server then transcodes the videoconference for provision toclients to include periodic key (e.g., I) frames, which as indicatedabove are able to cope with losses in transmissions. Additionally, thekey frames allow for seeking (e.g., allows the user to skip overportions of the videoconference when viewing) and allows new viewersjoin a session at any time (although new sessions could be formed at anytime otherwise). In this embodiment, the transcoding process performedby the recording server results in a lower number of possible streamingclients at any one time. This embodiment also reduces the number ofrecording sessions that the recording server can perform at any onetime.

A more preferred embodiment is shown in FIG. 8B. In FIG. 8B, the layoutprovided to both the other endpoints and the recording server does notinclude the recording server as a participant. Accordingly, the screenlayout is not reduced by having to show the recording server.Additionally, participants of the videoconference may be aware thatrecording is occurring by a recording indication in the layout (shown inthe upper left hand corner). In some embodiments, this recordingindication may only be displayed to the participants, but may not besent to the recording server. As also shown, the MCU may transmit thevideoconference information to the recording server differently thanthat to the videoconference endpoints, in this case including theperiodic iframes. The recording stream may be unique in other manners,as indicated above. The recording server can then rebroadcast the sourcestream without transcoding it and can therefore support a larger numberof concurrent recording and streaming sessions.

FIG. 9 illustrates an exemplary interface (e.g., on a web page) forviewing a recorded videoconference. As shown, the user may be able toview the recorded videoconference, seek any point in thevideoconference, adjust volume, screen size, etc. Additionally, theinterface may indicate metadata of the videoconference, including, forexample, the owner or creator, the record date, the duration, theplayback bandwidth, the channel, any tags, etc. Additionally, the ownermay be able to edit this information, delete the recording, feature thevideo, download the video, view the recording log, etc. Further viewersmay be able to post comments about the video or browse to view othervideos (e.g., looking at other channels, recent videos, featured videos,etc.). Thus, FIG. 9 illustrates an exemplary interface for viewingrecorded videoconferences. The interface may be hosted by the recordingserver or another server, as desired.

In further embodiments, the MCU and/or endpoint and/or the recordingserver may be configured to use speech to text to transcribe thevideoconference, e.g., into English or translated to other languages(assuming the videoconference is performed in English). Accordingly,this text may be overlaid on the video (or incorporated into the video)when provided to the recording server and/or the client (depending onwhether the endpoint or the recording server performs thetranscription). Additionally, the text may be searchable, e.g., by theclient. Accordingly, the client may search for specific text in thevideoconference. The search results may return in various manners. Forexample, in one embodiment, the client may present a text transcriptionof the videoconference and the user may view the transcription at theportion that has the searched text. Additionally, or alternatively, thetext may be associated with specific time frames of the videoconference(e.g., when those words were spoken) and the user may be able to skip tothat portion of the videoconference for viewing. For example, a user maybe able to select a search result and the recording may be automaticallyplayed at the portion of the recording that corresponds to the searchresult. In further embodiments, users may be able to search across aplurality of recordings (e.g., all recordings on a server) to find arecording whose transcript contains specified text (or otherwise matchesthe search criteria). In such embodiments, a list of matching recordingsmight be shown, and results within each recording may be provided in themanner described above, although other implementations are envisionsed.Additionally, the generated transcripts may be edited by users, e.g.,those users who have access to perform editing, such as the creator ofthe videoconference or other participants.

Thus, in some embodiments, additional features may be available to usersviewing the recorded videoconference, e.g., after the videoconferencehas been performed.

FIG. 10—Recording a Videoconference Using Configurations

FIG. 10 illustrates a method for recording a videoconference usingconfigurations. The method shown in FIG. 10 may be used in conjunctionwith any of the computer systems or devices shown in the above Figures,among other devices. In various embodiments, some of the method elementsshown may be performed concurrently, performed in a different order thanshown, or omitted. Additional method elements may also be performed asdesired. As shown, this method may operate as follows.

In 1002, a videoconference may be initiated or performed, similar to 602above. As already described, one of the videoconferencing endpoints orunits may act as an MCU for the videoconference and may be coupled to arecording server.

In 1004, one or more configurations may be selected for recording thevideoconference. These configurations may have been previously createdby a user, e.g., before the videoconference is initiated in 1002. Theconfiguration may specify any of various parameters for recording thevideoconference. For example, the configuration may specify the qualityof the recording, resolution, streaming bitrate, encoding format, framerate, how often iframes should be generated, audio/video codec, aspectratio, and/or recording metadata, such as recording description(s),ownership information, viewing authorization, and recordingcategorization (e.g., a channel to which the recording should be addedor storage directory). The configuration may also indicate if thevideoconference should be recorded and/or streamed live. Further, theconfiguration may specify a recording server, e.g., the address of therecording server that should be used. However, in further embodiments,the address of the recording server may be stored or configuredseparately, e.g., on a per videoconferencing endpoint or unit basis.

Additionally, configurations may be associated with any of a variety ofentities. For example, a configuration may be associated with respectiveusers or participants, with a particular videoconferencing location orroom, with an videoconferencing endpoint or unit, etc. For example,video endpoints can be preconfigured with configurations (sometimesreferred to as “keys” or “pins”), e.g., which may enable the singletouch recording described herein. The term “pin” here may refer to a pinnumber or other identification code used to identify a previouslycreated configuration. In one embodiment, an endpoint owned by a groupof users can be preconfigured for one touch recording to record orstream content that is access restricted to the group, e.g., just themembers of the group, to the highest access level of the group, to thelowest access level of the group, etc.

There may be more than one configuration associated with a given entity.For example, in some embodiments, users may have or own multiplerecording configurations. This may allow them to create recordings andlive streams with different recording quality or viewing authorizations.As a specific example, the CTO of a company may have a plurality ofdifferent associated configurations, e.g., one for management, one forengineering, one for sales, one for legal, etc. As indicated above,these configurations may specify different viewing rights or locationsfor publishing, among other possible differences. Further, a user mayhave more than one key to specify the different bitrates or codecs thatshould be generated for videoconferences, e.g., to support his differentdevices. More specifically, the user may want to support his mobiledevice and computer, and may have a different configuration for each.Both configurations may be used for a given videoconference, so thateither device could view the videoconference, if desired.

FIG. 11 illustrates an exemplary screen shot of an interface formanaging configurations (shown as “recording keys”). In this embodiment,the user “administrator” has an associated configuration “6512”. Thisconfiguration has an associated name (“1 Mbps”), channel (“main”, whereassociated videoconferences may be published), viewing rights(“everyone”), whether it is recorded (“yes”), whether it is streamedlive (“yes”), a description (left blank), associated tags (“demo at 1Mbps”), bitrate (“1024”), and presentation bandwidth (“70”). Note thatthe information stored in this configuration is exemplary only and othersettings and fields are envisioned.

In some embodiments, the configurations may be selected before or duringthe videoconference, as desired. For example, the configurations may beselected prior to the videoconference, prior to recording (e.g., butduring the videoconference), or at the time the recording is initiated.Additionally, the configurations may be fully specified before selectionor may be specified during selection, e.g., when recording is invoked.

Further, the configurations may be selected automatically (e.g., by acomputer system (e.g., the videoconferencing unit) without user inputspecifying the configurations) and/or manually. For example, for manualselection, one or more of the participants may select theconfiguration(s) using a remote or other input device. Note that themanual selection may be performed in addition to automatic selection, ifdesired.

For automatic selection, the MCU, endpoint(s) or recording server may beconfigured to perform the selection. In a simplistic case, the MCU mayuse default configuration(s), e.g., associated with the MCU and/or theendpoints of the videoconference. However, the automatic selection maybe more sophisticated. For example, when a recording is initiated, theMCU, one of the videoconferencing units, and/or the recording server maybe configured to use voice/face recognition to identify participants ofthe videoconference. For example, in one embodiment, the voice and/orface recognition may be based on videoconferencing information receivedfrom the endpoint that originated the recording to identify theparticipant(s) who initiated the recording. Alternatively, theparticipants of the videoconference may already be known (e.g.,specified in a videoconference request or call) or may be specified bythe participants, as desired. Accordingly, the configuration(s)associated with those participant(s) may be selected.

The endpoint (e.g., from the recorder) can offer to start arecording/streaming session using one of the recording user's profiles.Following this example, the recording's ownership can be assigned to theuser initiating the recording. This allows users to record/streamwithout having to remember recording configuration(s). However, theselected configurations could be based on other participants of thevideoconference instead of or in addition to the initiating participant,as desired.

In 1006, the conference may be recorded based on the one or moreconfigurations specified in 1004. The conference may be recorded similarto the description above regarding 606. However, the recording servermay record or store the videoconference based on the selectedconfigurations. For example, the recording may be created using theconfiguration or a profile associated with the configuration (e.g.,which may have been preconfigured). In one embodiment, thestreaming/recording server may store multiple recording profiles thatspecify streaming bitrate, codecs and recording metadata. Recordingmetadata may include a recording description, ownership information,viewing authorization and recording categorization (channel), amongother possibilities. A profile may also identify if the content shouldbe recorded and/or streamed live. Each recording profile may beassociated with a configuration (e.g., a key or pin) described above. Insome embodiments, video endpoints can download recording profiles fromthe recording server for a key entered by a user and generate theappropriate stream.

In 1008, the recorded videoconference may be provided to one or moreclients, similar to 608 described above.

Embodiments described above may also apply to streaming videoconferences(e.g., instead of recording videoconferences). In such embodiments,similar configurations may be used (except those related to the storedrecordings) and may be referred to as “streaming configurations” ratherthan “recording configurations”.

FIG. 12—Recording a Videoconference in Response to a Single UserInteraction

FIG. 12 illustrates a method for recording a videoconference in responseto a single user interaction. The method shown in FIG. 12 may be used inconjunction with any of the computer systems or devices shown in theabove Figures, among other devices. In various embodiments, some of themethod elements shown may be performed concurrently, performed in adifferent order than shown, or omitted. Additional method elements mayalso be performed as desired. As shown, this method may operate asfollows.

In 1202, a videoconference may be initiated or performed among aplurality of videoconference endpoints, as described in 602 above.

In 1204, a request to record the videoconference may be received inresponse to a single user interaction. For example, the single userinteraction may be one of the participants, e.g., at an endpoint,utilizing a remote device, e.g., to press a record button. The remotedevice may include a remote control, a touch device, a speakerphone,etc. For example, the user may select a record button that is dedicatedto initiating recording of the videoconference. In further embodiments,the user may use a single user interaction provided by applicationsrunning on devices, e.g., on touch devices such as mobile phones ortablet devices, on netbooks or laptops, etc.

Other single user interactions are envisioned. For example, aparticipant may use a voice command to initiate the recording.Alternatively, or additionally, the participant may use a laptop toprovide a command to the endpoint or recording server to initiate therecording. In one embodiment a gesture may be used (e.g., on a touchscreen or provided to a camera of the videoconference endpoint, etc.).The participant may use a mouse to select a button to initiate therecording. Regardless of the method used, only a single interaction ofthe user may be required to initiate the recording of thevideoconference. For example, the participant may not have to go througha tedious set of inputs to initiate the recording, but may simpleprovide one interaction or input to initiate the recording of thevideoconference. Additionally, the participant may not need to specify arecording configuration in order to initiate the recording of thevideoconference.

In 1206, recording of the videoconference may be initiated in responseto the request in 1204. The recording of the videoconference may beperformed as described herein. Note that no further user interaction maybe required to initiate (or even to perform) the recording other thanthe single user interaction of 1204. Thus, in one embodiment, theparticipant may only provide the single user interaction and no furtherinteraction may be required for the recording to begin. The recordingmay be performed utilizing one or more pre-stored recordingconfigurations, such as those described above, among others. Forexample, the pre-stored recording configurations may be the recordingconfigurations described in FIG. 10. More specifically, one of thepre-stored configurations used to record the videoconference may beassociated with a participant of the videoconference, an endpoint of thevideoconference, etc.

The recording configurations may be automatically selected, e.g., by theendpoint receiving the single user interaction, by the MCU, and/or bythe recording server. For example, the recording configurations may beautomatically selected by using configurations associated withparticipants in the videoconference, e.g., using one or more recordingconfigurations associated with the participant providing the input toinitiate recording, associated with the endpoint that receives theinput, associated with the participant that requested or initiated thevideoconference, associated with the highest ranking participant in thevideoconference, associated with the recording server (e.g., a defaultrecording configuration), etc. In this embodiment, the participant maynot provide any input other than the single user interaction to performthe recording. For example, the participant may not perform any actionprior to or after the single user interaction in order to initiateand/or perform the recording of the videoconference.

Alternatively, or additionally, the recording configurations may includeany configurations that participants in the videoconference (or otherusers) selected prior to initiating the recording, e.g., selected priorto or during the videoconference, but before providing the single userinteraction. In one embodiment, participants may specify desiredrecording configurations when the videoconference is initiated, when theparticipant accepts an invite to the videoconference, during scheduling,etc. In one embodiment, a first participant may select one or moreconfigurations and a second participant may provide the single userinteraction to record the videoconference, e.g., without specifying anyconfigurations.

The method of 1200 may be performed by any of a variety ofvideoconference equipment. For example, 1204 and 1206 may be performedby an MCU, an endpoint acting as an MCU, or any endpoint of thevideoconference, as desired. For example, the request may be provided asuser input to the MCU or endpoint. Alternatively, 1204 and 1206 may beperformed by a recording server in communication with one or more of thevideoconference MCU/endpoints. For example, the request may be providedas a message to the recording server to initiate the recording, e.g.,from an MCU or endpoint of the videoconference.

FIG. 13—Recording a Videoconference with a Separate Presentation Stream

FIG. 13 illustrates a method for recording a videoconference withseparate participant and presentation streams. The method shown in FIG.13 may be used in conjunction with any of the computer systems ordevices shown in the above Figures, among other devices. In variousembodiments, some of the method elements shown may be performedconcurrently, performed in a different order than shown, or omitted.Additional method elements may also be performed as desired. As shown,this method may operate as follows.

In 1302, a videoconference may be initiated or performed among aplurality of videoconference endpoints, as described in 602 above.

In 1304, first videoconference information may be received forrecordation. The first videoconference information may correspond toparticipants in the videoconference. The first videoconferenceinformation may be received by a recording server, e.g., from an MCU ofthe videoconference and/or an endpoint of the videoconference, asdesired. As used herein, videoconference information that “correspondsto participants in the videoconference” (or “participant videoconferenceinformation”) refers to video and/or audio that generally include theparticipants. For example, video that corresponds to participantsgenerally includes video of participants at a correspondingvideoconference location. While the participant(s) may not always bedisplayed, e.g., when the participant steps away from the camera, theprimary purpose of the video is for displaying images of theparticipant(s). Thus, the video that corresponds to participants mayinclude an image of at least one participant for more than 50% of thetime. Note that the percentage may typically be higher, e.g., 70%, 80%,90%, 95%, 100%, etc.

In 1306, second videoconference information may be received forrecordation. The second videoconference information may be received by arecording server, e.g., from an MCU of the videoconference and/or anendpoint of the videoconference, as desired. The second videoconferenceinformation may correspond to auxiliary video of the videoconference.The second videoconference or auxiliary videoconference information maybe any videoconference information that does not correspond to theparticipants of the videoconference. For example, the secondvideoconference information may correspond to a presentation presentedduring the videoconference (e.g., may be “presentation videoconferenceinformation”). For example, the second videoconference information mayinclude video of slides of a presentation as one of the participantsdiscusses the various slides. The presentation video may further includevideo of the participant interacting with the slides (e.g., using amouse to point at various portions within a slide, highlighting variousportions, etc.). In one embodiment, the presentation video may includevideo provided from a participant's computer, e.g., as the participantexecutes programs on the computer. In alternated embodiments, thepresentation video may include video of a presentation provided via aprojection machine or other mechanism. However, as indicated above, thesecond videoconference information may not be presentation information,but may generally be videoconference information that does notcorrespond to participants of the videoconference, since thatinformation is already received in 1304.

In some embodiments, the second videoconference information may not bepresent for a portion of the videoconference. For example, one of theparticipants may present a presentation during the videoconference forless than the entirety of the videoconference (e.g., for 5 minuteswithin the 30 minute videoconference). Note that the secondvideoconference information may be present a plurality of differenttimes throughout the videoconference. For example, the videoconferencemay begin and participants of the videoconference may discuss variousmatters for a first time period, and no second videoconferenceinformation may be present during the first time period. At a secondtime period, a first participant may present a presentation. At a thirdtime period, after the first participant has finished providing thepresentation, and no second videoconference information may be presentduring the third time period. At a fourth time period, a secondparticipant may provide video from his computer, e.g., to illustrate aprogram or graph that is pertinent to the conversation. Thus, in thisparticular example, the second videoconference may be present during thesecond period and the fourth period, but not during the first and thirdperiods. Additionally, it should be noted that the secondvideoconference information may be of different natures, where there isa plurality of different types of second videoconference information.For example, each different second videoconference information may notbe a presentation or of the same type of videoconference information(e.g., one may correspond to a presentation while another may correspondto a different type of second videoconference information).

In 1308, a timeline of the second videoconference information may bestored for recording the videoconference. The timeline may specify whenthe second videoconferencing information was present in thevideoconference. The timeline may be received by a recording server,e.g., from an MCU of the videoconference and/or an endpoint of thevideoconference, as desired. Alternatively, or additionally, thetimeline may be generated by the recording server, e.g., as the firstand second videoconference information is received from the MCU orendpoint of the videoconference. For example, the recording server maystore the times when the received second videoconference informationbegins and ends during the videoconference, e.g., relative to thebeginning of the videoconference (when the first videoconferenceinformation is initially received).

In 1310, a recording of the videoconference may be provided to one ormore clients. The recording of the videoconference may supply the firstvideoconference information, the second videoconference information,and/or the timeline of the second videoconference information in asingle stream or package or may provide them separately as desired. Forexample, the recording of the videoconference may include only the firstvideoconference information at a first time (e.g., at the beginning ofthe videoconference). The timeline may be provided separately (e.g.,from the first videoconference information or the recording) or may beprovided with the first videoconference information. Later, the secondvideoconference information may be provided (e.g., when the secondvideoconference information was originally present in thevideoconference). This second videoconference information may beprovided with the first videoconference information in the recording ormay be provided separately, as desired. In one embodiment, the first andsecond videoconference information may be provided initially, even ifthe second videoconference information is not present in thevideoconference until a later time. Accordingly, the client may be ableto buffer or store the second videoconference information and beginplayback of the second videoconference information at a later time,based on the received timeline of the second videoconferenceinformation.

Similar to descriptions above, the recording (e.g., the firstvideoconference information and/or the second videoconferenceinformation) may be formatted or provided to the clients withoutsubstantial modification since being received in 1304 and 1306. Also,similar to embodiments above, the first videoconference information andsecond videoconference information may be different than thevideoconference information provided among the plurality of endpoints inthe videoconference. For example, the original videoconferenceinformation (whether it be participant videoconference information orauxiliary videoconference information) provided to endpoints may notinclude periodic iframes or may not be in the streaming friendly formatof the first and second videoconference information. However, in otherembodiments, the first and second videoconference information may betranscoded or modified before being provided to the clients, as desired.

In 1312, the one or more clients may playback the recording of thevideoconference. More specifically, the one or more clients may use thetimeline to appropriately display the second videoconference informationduring playback of the recording of the videoconference. The firstvideoconference information may be played back for the duration of theplayback of the recorded videoconference.

In one embodiment, each client may provide the first videoconferenceinformation for playback on the display of the client, determine a firstpoint in the videoconference when a portion of the secondvideoconferencing information begins using the timeline, and provide thesecond videoconference information for playback on the display at thefirst point. Further, each client may determine a second point in thevideoconference when the portion of the second videoconferencinginformation ends using the timeline, and cease providing the secondvideoconference information for playback on the display at the secondpoint. This process may be repeated a plurality of times for each timeperiod that the second videoconference information is present.

FIG. 14—Recording a Videoconference Including Boundary Information

FIG. 14 illustrates a method for recording a videoconference withboundary information of a composite image. The method shown in FIG. 14may be used in conjunction with any of the computer systems or devicesshown in the above Figures, among other devices. In various embodiments,some of the method elements shown may be performed concurrently,performed in a different order than shown, or omitted. Additional methodelements may also be performed as desired. As shown, this method mayoperate as follows.

In 1402, a videoconference may be initiated or performed among aplurality of videoconference endpoints, as described in 602 above.

In 1404, the videoconference may be recorded, according to any of theembodiments described herein. The recorded videoconference may includecomposite images or may be a composite video. As discussed above, thecomposite images may include a plurality of participant images inrespective portions of the composite image. However, it should be notedthat some portions of the composite images may not correspond toparticipants, but may be other video, such as presentations or othervideo. Thus, the recorded videoconference may include audio/video of theparticipants, audio/video of a presentation, etc.

In 1406, boundary information of the videoconference may be stored whichindicates the boundary of the composite images in the recordedvideoconference. Note that the boundary information may be stored aspart of the recorded videoconference or may be stored separately, asdesired. Additionally, the boundary information may be received as partof, or separately from, the videoconference information used to recordthe videoconference. For example, an endpoint may provide thevideoconference information to record the videoconference and may alsoprovide the boundary information of the videoconference information(e.g., which may be included in the videoconference information orprovided separately, as desired).

In 1408, the recorded videoconference and boundary information may beprovided to one or more clients. Each client may execute software (e.g.,a videoconference or video player) to playback the recording to one ormore users of the client. The client may be any of various devices, suchas a computer, a mobile device, a tablet, a mobile phone, etc.

In 1410, a client may use the boundary information to change the layoutof playback of the recorded videoconference. For example, the client maybegin playback of the recorded videoconference according to a firstlayout (e.g., the layout that the videoconference was originallyrecorded in). During playback, a user of the client may request a changein layout of the videoconference. For example, the user may select a newdesired layout from a plurality of possible layouts, e.g., which may bedisplayed in a graphical manner on the client display. Alternatively, oradditionally, the user may select a “change layout” option and theclient may cycle from a current layout to a new layout, e.g., in acyclical order. In further embodiments, the user may be able to manuallyselect portions of the layout and move, resize, or delete them. Forexample, the user may drag a border within the composite image toincrease or decrease the relative sizes of portions within the layout.Alternatively or additionally, the user may select one of the portionsand choose various options for that portion, e.g., increase size,decrease size, maximize size, minimize size, remove, split, etc.

In some embodiments, the layout of the playback of the recording maychange automatically. For example, if one of the participants in thevideoconference begins a presentation and new presentation data isprovided for display, the client may automatically change the layout ofthe playback of the recording in order to provide the presentation videofor display. Similarly, when the new presentation ends, the client mayautomatically change the layout of the playback of the recording toremove the presentation from being displayed (e.g., thereby reallocatingthe presentations portion to other video). Additionally, the layout ofthe playback of the recording may change as new users are added orremoved, as the active talkers in the videoconference change, etc.

Accordingly, the client may use the provided boundary information tochange the layout of the recorded videoconference in response to theuser's input. Note that this change in layout may not change theunderlying recording, but instead may simply change the presentation ofthe recorded videoconference for that particular client. Moreparticularly, the change in layout may be performed locally by theclient and not change any data of the recorded videoconference.

In addition to the boundary information, transcript information of thevideoconference may be stored and/or provided to the one or moreclients. In one embodiment, the one or more clients may be configured toutilize the transcript as subtitles of the recorded videoconference orfor searching the videoconference. For example, a user of the client(s)may search for a portion of the videoconference using keywords. Inresponse, the client may search the transcript for those keywords andallow the user to skip to the corresponding portion in thevideoconference. As one example, the user may search for the word“soccer” and the client may move playback of the recordedvideoconference to a portion of the videoconference where one of theparticipants said the word “soccer”. The client may also provide a listof search results and the user may be able to select each instance ofthe results to view the corresponding portion of the recordedvideoconference.

FIG. 15—Recording a Videoconference According to Connection Quality

FIG. 15 illustrates a method for recording a videoconference based onconnection quality. The method shown in FIG. 15 may be used inconjunction with any of the computer systems or devices shown in theabove Figures, among other devices. In various embodiments, some of themethod elements shown may be performed concurrently, performed in adifferent order than shown, or omitted. Additional method elements mayalso be performed as desired. As shown, this method may operate asfollows.

In 1502, a videoconference may be initiated or performed among aplurality of videoconference endpoints, as described in 602 above.

In 1504, a request to record the videoconference may be received. Therequest may be received from a user, e.g., by an endpoint of thevideoconference. For example, the request may include the user providinginput to request to record the videoconference Alternatively, oradditionally, the request may be received from a recording server froman endpoint or MCU of the videoconference. For example, the recordingserver may receive a message requesting to record the videoconferencefrom the MCU of the videoconference. The request may generally begenerated or received according to any of the embodiments describedabove, e.g., in with respect to FIG. 6.

In 1506, a videoconference device (e.g., an endpoint and/or MCU) havinga high quality connection to the recording server may be determined.This determination may be performed by the recording server (e.g., bythe recording server measuring connection quality between each endpoint)and/or by each videoconference device participating in thevideoconference. Generally, the determination of the high qualityconnection may be determined by any involved device.

In one embodiment, the method may determine the high quality connectionby determining which videoconference device(s) are on the same localarea network as the recording server. For example, the method maygenerally assume that any local area network connection is of higherquality than wide area network connections (e.g., Internet connections).Alternatively, or additionally, the high quality connection may bedetermined by detecting packet loss or bandwidth available between thevideoconference device(s) and the recording server. For example, theconnection may be determined as being “high quality” if the packet lossis below a threshold, if the bandwidth exceeds a threshold, etc.Generally, the videoconference device having the highest qualityconnection (e.g., according to whichever metric that is being used) maybe the determined videoconference device for providing videoconferenceinformation of the videoconference to the recording server forrecording.

However, it should be noted that there may be a determination ofcapacity that has a higher or lower priority than the connectionquality. For example, the connection quality may only be determined forvideoconference devices that have the processing or bandwidth capacityto provide the videoconference information for recording. Alternatively,the connection quality may be determined for all availablevideoconference devices and the capacity of the videoconference devicesmay be used to break ties between videoconference devices having similarconnection qualities.

Where a high quality connection is not determined, the highest availableconnection quality of the available videoconference devices may be used.The highest quality connection may be determined according to themetrics described above, e.g., based on packet loss, bandwidth, etc.,among other possible metrics.

Additionally, or alternatively, the videoconference device may bedetermined according to which device requires the least (or is among asubset that requires a lower amount of) effort in traversing afirewall/NAT setup.

In 1508, the determined videoconference device may providevideoconference information for recording the videoconference to therecording server. The videoconference information received in 1610 maybe similar to the videoconference information received for recordingdescribed above, e.g., in 606 above. For example, the videoconferenceinformation may be “streaming friendly” and require little or nomodification for provision to clients (at least for a first recordingformat), or not, as desired.

By determining and using a connection with high quality, the quality ofthe recording may be improved. For example, recording a videoconferencewhen using a connection with less quality, the quality of recording maybe affected by bandwidth fluctuations and packet loss. It should benoted that the recording server may generally have the ability tore-request portions of a videoconference that have been affected byconnection issues, and so a minimum level of quality may be maintainedvia this mechanism. Further, various forward error correction proceduresmay be used to reduce packet loss. However, it is still generallypreferred that a high quality connection be used to avoid thecomplication of such procedures.

Note that the determination of the high quality connection may bedetermined a plurality of times (e.g., periodically) during thevideoconference. For example, the method may determine if the currentvideoconference device still has a high quality connection and/or ifanother videoconference device has a better connection. If theconnection quality has sufficiently deteriorated or if anotherconnection is sufficiently better, the provision of the videoconferenceinformation in 1508 may be changed to another videoconference device.Alternatively, or additionally, this update of the determination may beperformed whenever the quality of connection of the videoconferencedevice drops to an unacceptable level, e.g., when the connection drops,packet loss exceeds a threshold, bandwidth is lower than a specifiedthreshold, etc.

Exemplary Embodiment According to the Method of FIG. 15

The following describes an exemplary embodiment according to the methodof FIG. 15.

A videoconference may be established between a plurality ofvideoconferencing endpoints. At some point prior to or during thevideoconference, a request to record the videoconference may bereceived. For example, a user may press a record button, or some otheraction may trigger recording (e.g., the recording may beginautomatically). In one embodiment, the endpoint that receives the userinput or otherwise is involved in initiating the recording process maysend a message to other videoconference devices capable of providingvideoconference information to the recording server. For example, themessage may provide details regarding the requested recording (e.g., anaddress of the recording server, desired recording configurations orformats, etc.). The message may also specify the quality of connectionbetween the endpoint and the recording server. For example, the qualityof connection may be specified as a “recording metric”.

In response, any videoconference device that has a better recordingmetric (e.g., and has the capacity to perform the recording activity)may provide a response to the initiating videoconference endpoint andperform the provision of videoconference information to the recordingserver. The new videoconference device may provide the videoconferenceinformation according to the specified recording configuration, asspecified by the initial message above, or a subsequent message once thenew videoconference device has been assigned the task.

The method may be extended to videoconference devices that are not ableto perform the provision of the videoconference information to therecording server. For example, a user of such a videoconference devicemay request recording of the videoconference, and that device mayoffload the provision of the videoconference information to anotherdevice that is capable of performing the task. As a specific example, acomputer videoconferencing application may be able to participate invideoconference, but may not be configured to provide thevideoconference information for recording. Accordingly, the task may beoffloaded to a dedicated videoconference endpoint instead, following thedescriptions above.

In addition to the above embodiment, the method may be extended. Forexample, if an MCU is involved in the videoconference, the MCU may takeover the recording.

FIG. 16—Recording a Videoconference According to a Plurality of Formats

FIG. 16 illustrates a method for recording a videoconference accordingto a plurality of formats. The method shown in FIG. 16 may be used inconjunction with any of the computer systems or devices shown in theabove Figures, among other devices. In various embodiments, some of themethod elements shown may be performed concurrently, performed in adifferent order than shown, or omitted. Additional method elements mayalso be performed as desired. As shown, this method may operate asfollows.

In 1602, a videoconference may be initiated or performed among aplurality of videoconference endpoints, as described in 602 above.

In 1604, a request to record the videoconference may be received. Therequest may specify a plurality of desired recording formats. Forexample, the request may specify a one or more different recordingconfigurations which may overall specify a plurality of desiredrecording formats for the recording of the videoconference.Alternatively, the recording formats may be specified manually by aparticipant of the videoconference (or other user), determinedautomatically by the recording server, MCU, or endpoint of thevideoconference, etc.

In 1606, one or more recording resources may be determined for recordingthe videoconference according to at least one of the plurality ofrecording formats. As used herein, a “recording resource” may be anydevice that is capable of recording a videoconference, e.g., based onreceived videoconference information. In some embodiments, thisdetermination may be performed by providing a request to variousvideoconference devices or other devices. For example, the recordingserver may send a request to various videoconference devicesparticipating in the videoconference (or other devices that are notparticipating in the videoconference) to record the videoconferenceaccording to one of the plurality of recording formats. In furtherembodiments, the request may be provided by other entities than therecording server, e.g., the MCU of the videoconference, an endpoint ofthe videoconference (e.g., the endpoint that initiated the recording,possibly in response to user input), etc. Recording resources withadequate processing capability may respond to the request to perform therecording of the videoconference. Alternatively, or additionally,recording may simply be assigned to one of the recording resources.

In some embodiments, similar to embodiments described above with respectto FIG. 15, the determination of the recording resource may be based onthe quality of connection between the recording resource and therecording server. Alternatively, or additionally, the determination ofthe recording resource may be based on the quality of connection betweenthe recording resource and the videoconference device providingvideoconference information of the videoconference (e.g., where thevideoconference information is provided directly to the recordingresource instead of through the recording server, as described below).In further embodiments, the determination of the recording resource maysimply be based on the quality of the videoconference information thatthe recording resource can receive, regardless of the source. Thus,various quality information (e.g., connection quality information) maybe used to determine the recording resource.

In 1608, the videoconference information of the videoconference may bereceived for recording. Note that 1608 may apply to both the recordingserver and the determined recording resources. As indicated above, thevideoconference information may be provided simultaneously to both therecording server and the determined recording resource, or it may beprovided in sequence, e.g., where the recording server receives thevideoconference information and then provides the videoconference to therecording resource so that the recording resource can generate a secondrecording of the videoconference in a second recording format.Alternatively, if the determined recording resource is participating inthe videoconference (e.g., if it is an endpoint), it may simply use thenormal videoconference information of the videoconference rather thanreceiving the videoconference information that is provided to therecording server in 1608.

In 1610, the videoconference information may be recorded according to afirst recording format to generate a first recording of thevideoconference. Note that, similar to embodiments described above, thevideoconference information received in 1608 may already besubstantially in the first recording format or may be in a format thatrequires transcoding in order to generate the recording in the firstrecording format, as desired. For example, the videoconferenceinformation received in 1610 may be already in substantially one of theplurality of formats, which may be the first recording format or adifferent recording format that the recording server may transcode intothe first recording format, as desired. The videoconference informationreceived in 1610 may be similar to the videoconference informationreceived for recording described above, e.g., in 606 above.

In 1612, a second recording may be received from the determinedrecording resources. The second recording may be recorded according to asecond recording format that is different from the first recordingformat. The second recording may be received as a packaged recording ormay be received as videoconference information (e.g., similar to 1608),but formatted or transcoded according to the second recording format (orsubstantially in the second recording format without requiringsubstantial modification of the videoconference information). Generally,the recording server may perform the bulk of the required transcodingsand this offloading of transcoding or recording may be performed whenthe recording server lacks capacity.

In 1614, the first and second recordings may be provided to first andsecond clients, e.g., simultaneously. For example, the recording server(or a separate streaming server) may receive a first request to view thevideoconference from a first client and may receive a second request toview the videoconference from a second client. The method may determinethat the first client requires the first recording format and the secondclient requires the second recording format. For example, the firstrequest may specify the first recording format and the second requestmay specify the second recording format. Alternatively, the formats maybe determined based on the clients themselves. For example, the methodmay determine that the first client is a mobile phone and provide therecording in a format that is appropriate for the device type of thefirst client (in this particular case, a lower resolution format may beselected). Similarly, the method may determine that the second client isa laptop computer and may provide a format that is appropriate forcomputers (e.g., one that is a higher resolution than the first format).Note that the provision of the first and/or second recordings may beperformed during the videoconference or after completion of thevideoconference, as desired. The offloading of transcoding or recordingmay be especially beneficial in embodiments where the first and secondrecordings are performed during the videoconference, since they cannotbe produced over time, as capacity allows. The provision of the firstand second recordings may be performed substantially in real time withrespect to the videoconference (e.g., with less than 1 second delay, 2second delay, 5 second delay, 10 second delay, 30 second delay, etc.).

In further embodiments, instead of, or in addition to the plurality offormats being determined by the request to record the videoconference orother devices recording or participating in the videoconference, thedesired recording formats may be selected based on requests receivedfrom clients to view the videoconference. For example, the recordingserver may offload recording or transcoding operations in response torequests from clients to view the recording in a format that is notcurrently available. Thus, the recording server may generate recordingsof the videoconference (e.g., itself or in a distributed manner) inresponse to requests from clients. The recording server may then providethe newly created recording to the client in response to the request. Insome embodiments, the recording server may be able to do thissubstantially in real time, such that the client may receive the newlycreated recording on demand. The recording server may perform this bystreaming the newly created recording as it is generated, whether it isbeing generated by the recording server or by another recordingresource. In the instance of another recording resource performing therecording generation, the new recording may be streaming from therecording resource to the recording server, which may in turn stream thenew recording to the client. Alternatively, the new recording may bestreamed directly from the recording resource to the client, as desired.

Exemplary Embodiment of the Method of FIG. 16

The following provides an exemplary embodiment of the method of FIG. 16.

A videoconference may be performed between three endpoints and one ormore of the endpoints may be able to record the videoconference.Additionally, one or more of the endpoints may have a high qualityconnection to a recording server (e.g., they may be on the same localarea network as the recording server).

A participant at one of the endpoints (e.g., endpoint “A”) may provideuser input to initiate the recording, e.g., by pressing a record button.The participant may or may not specify a recording configuration (e.g.,using a PIN or KEY, or via other mechanisms). As indicated above, therecording configuration may specify which channel the recording shouldbe published to, whether it should be capable of being streamed or not,viewing rights required to access the recording, etc. The recordingconfiguration may also specify all of the formats that should beavailable, e.g., while the videoconference is being performed (e.g., forlive streaming of the recording). In some embodiments, this informationmay be referred to as “transcode details”.

Endpoint A may examine the recording configuration and/or transcodedetails. Endpoint A, or another videoconference device may then pollother endpoints in the videoconference to determine which endpoint isable to perform the extra transcoding work needed to be done for therecording. In one embodiment, each other endpoint in the videoconferencemay be able to transcode a single format of recording responsibility,and the remainder of the transcoding may be performed by the recordingserver, although other embodiments are envisioned. Additionally, otherendpoints (or other recording resources) that are not participating inthe videoconference may be used to perform the transcoding work, asdesired. However, endpoints that are not capable of performing theadditional work, or who otherwise do not wish to take up the additionalload, may not respond to the query from endpoint A.

Finally, the endpoints may perform the transcoding according to thedetermination above and provide the properly formatted recording orvideoconference stream to the recording server. The recording server maythen stream the plurality of formatted recording to clients, e.g.,during or after the videoconference. As indicated above, in someembodiments, the streaming may be performed substantially in real timewith respect to the videoconference.

Further Embodiments

In further embodiments, the endpoint and/or recording server can alsouse face and/or voice recognition on videos from all participatingendpoints to determine participant names. The recording can be taggedwith the participant information determined to enable search queriesbased on participants. The recorder can also offer to store therecording with viewing privileges restricted to the group of callparticipants. Thus, image and voice profiles can be used to identifyparticipants in recordings and live streams. Image and voice recognitioncan augment each other to increase the accuracy of determining a user ina video.

In some embodiments, the endpoint and recording server may generatevoice and facial profiles to enable the automatic detection ofparticipants (e.g., for selecting configurations) described above. Forexample, sections of video recordings on the recording server can betagged with the name of the active speaker. This information frommultiple recordings can be used to create a voice and/or face profilefor the user. In further embodiments, the recording server can storeuploaded photos of users in their user profiles. Recording viewers canalso pause recordings and mark out users in the recording (e.g., to tagparticipants of the videoconference). Multiple uploaded images andtagged images from recordings can be used to create an image profile ofthe user.

Additionally, it should be noted that any of the embodiments describedabove regarding recording or recordings of a videoconference may beextended to streaming the videoconference (e.g., without recording thevideoconference). For example, instead of recording a videoconferenceand providing the videoconference to clients, the videoconference may beprovided to a streaming server and the streaming server may provide therecording to clients in addition to or without recording thevideoconference. Further, embodiments described above may be modifiedfor instances for broadcasting from a single videoconferencing location,e.g., where a user would like to record or stream only his video/audioto a plurality of clients. Thus, instead of recording a videoconference,a user may record and/or broadcast information from a videoconferencingendpoint.

Advantages over Prior Systems

The above described embodiments may have numerous benefits over priormethods for recording videoconferences. For example, previous recordingsystems typically required a recording device to participate in thevideoconference, thereby taking a port of the MCU as well as valuablescreen resources on each other participant's screen. In embodimentsdescribed above, the video stream sent to the recording server is sentindependently, and does not involve the recording server acting as aparticipant or use in the composite image sent by the endpoint.Additionally, the recording server may not receive any video orcomposite section that corresponds to the recording server, andtherefore also does not have to remove itself from the recordedvideoconference.

Further, in embodiments described above, the audio and video sent to therecording server may be independent from any of the streams sent to theother participants. For example, the audio and video sent to therecording server may be unique or different from any of the audio andvideo sent to the other endpoints of the videoconference, unlike priorsystems.

Additionally, in embodiments where the endpoint transmits the video tothe recording server in a “streaming friendly” manner (e.g., includingperiodic i-frames), the recording server may retransmit the video of thevideoconference without having to perform inefficient transcodingprocedures before transmission, which has been previously required. Suchfunctionality allows the recording server to transmit thevideoconference information to many more clients (at the same time) thanwas previously possible.

Embodiments of a subset or all (and portions or all) of the above may beimplemented by program instructions stored in a memory medium or carriermedium and executed by a processor.

In some embodiments, a computer system at a respective participantlocation may include a memory medium(s) on which one or more computerprograms or software components according to one embodiment of thepresent invention may be stored. For example, the memory medium maystore one or more programs that are executable to perform the methodsdescribed herein. The memory medium may also store operating systemsoftware, as well as other software for operation of the computersystem.

Further modifications and alternative embodiments of various aspects ofthe invention may be apparent to those skilled in the art in view ofthis description. Accordingly, this description is to be construed asillustrative only and is for the purpose of teaching those skilled inthe art the general manner of carrying out the invention. It is to beunderstood that the forms of the invention shown and described hereinare to be taken as embodiments. Elements and materials may besubstituted for those illustrated and described herein, parts andprocesses may be reversed, and certain features of the invention may beutilized independently, all as would be apparent to one skilled in theart after having the benefit of this description of the invention.Changes may be made in the elements described herein without departingfrom the spirit and scope of the invention as described in the followingclaims.

What is claimed is:
 1. A non-transitory computer-readable memory mediumcomprising program instructions for recording a videoconference, whereinthe program instructions execute during a videoconference, wherein theprogram instructions are executable to: provide first videoconferenceinformation to a recording server for recording during thevideoconference, wherein the first videoconference informationcorresponds to participants in the videoconference, and wherein saidproviding the first videoconference information begins at a first time;provide second videoconference information to the recording server forrecording during the videoconference, wherein the second videoconferenceinformation corresponds to a presentation presented during thevideoconference, wherein the second videoconference information isprovided to the recording server during the provision of the firstvideoconference information, and wherein said providing the secondvideoconference information begins after the first time; provide atimeline of the second videoconferencing information to the recordingserver for recording, wherein the timeline specifies when the secondvideoconferencing information was present in the videoconference;wherein the timeline is used during playback of the recordedvideoconference to indicate when the second videoconference informationis available for display.
 2. The non-transitory computer-readable memorymedium of claim 1, wherein the first videoconference information, thesecond videoconference information, and the timeline are configured forrecording by the recording server.
 3. The non-transitorycomputer-readable memory medium of claim 1, wherein the firstvideoconference information and the second videoconference informationare configured for provision to clients by the recording server withoutsubstantial modification.
 4. The non-transitory computer-readable memorymedium of claim 1, wherein the program instructions are furtherexecutable to: provide third videoconference information to theplurality of videoconferencing endpoints, wherein the thirdvideoconference information is different from the first videoconferenceinformation.
 5. The non-transitory computer-readable memory medium ofclaim 4, wherein the first videoconference information comprisesperiodic iframes, wherein the third videoconference information does notcomprise periodic iframes.
 6. A method for recording a videoconference,comprising: providing first videoconference information to a recordingserver for recording during the videoconference, wherein the firstvideoconference information corresponds to participants in thevideoconference, and wherein said providing the first videoconferenceinformation begins at a first time; providing second videoconferenceinformation to the recording server for recording during thevideoconference, wherein the second videoconference informationcorresponds to a presentation presented during the videoconference,wherein the second videoconference information is provided to therecording server during the provision of the first videoconferenceinformation, and wherein said providing the second videoconferenceinformation begins after the first time; providing a timeline of thesecond videoconferencing information to the recording server forrecording, wherein the timeline specifies when the secondvideoconferencing information was present in the videoconference;wherein the timeline is used during playback of the recordedvideoconference to indicate when the second videoconference informationis available for display.
 7. The method of claim 6, wherein the firstvideoconference information, the second videoconference information, andthe timeline are configured for recording by the recording server. 8.The method of claim 6, wherein the first videoconference information andthe second videoconference information are configured for provision toclients by the recording server without substantial modification.
 9. Themethod of claim 6, further comprising: providing third videoconferenceinformation to the plurality of videoconferencing endpoints, wherein thethird videoconference information is different from the firstvideoconference information.
 10. The method of claim 9, wherein thefirst videoconference information comprises periodic iframes, whereinthe third videoconference information does not comprise periodiciframes.
 11. A non-transitory computer-readable memory medium comprisingprogram instructions for recording a videoconference, wherein theprogram instructions execute during a videoconference, wherein theprogram instructions are executable to: receive first videoconferenceinformation during the videoconference, wherein the firstvideoconference information corresponds to participants in thevideoconference, and wherein said receiving the first videoconferenceinformation begins at a first time; receive second videoconferenceinformation during the videoconference, wherein the secondvideoconference information corresponds to a presentation presentedduring the videoconference, wherein the second videoconferenceinformation is received during the reception of the firstvideoconference information, and wherein said receiving the secondvideoconference information begins after the first time; receive atimeline of the second videoconferencing information, wherein thetimeline specifies when the second videoconferencing information waspresent in the videoconference; wherein the timeline is used duringplayback of the videoconference to indicate when the secondvideoconference information is available for display.
 12. Thenon-transitory computer-readable memory medium of claim 11, wherein saidreceiving the first videoconference information, said receiving thesecond videoconference information, and said receiving the timeline isperformed by a recording server to record the videoconference.
 13. Thenon-transitory computer-readable memory medium of claim 11, wherein theprogram instructions are further configured to: provide the firstvideoconference information, the second videoconference information, andthe timeline of the second videoconference information to one or moreclients, wherein the one or more clients are configured to use thetimeline during playback to present the second videoconferenceinformation.
 14. The non-transitory computer-readable memory medium ofclaim 11, wherein said receiving the first videoconference information,said receiving the second videoconference information, and saidreceiving the timeline is performed by a client to playback thevideoconference.
 15. The non-transitory computer-readable memory mediumof claim 11, wherein the program instructions are further executable toprovide the videoconference for playback on a display, wherein saidproviding comprises: providing the first videoconference information forplayback on the display; determining a first point in thevideoconference when a portion of the second videoconferencinginformation begins using the timeline; and providing the secondvideoconference information for playback on the display at the firstpoint.
 16. The non-transitory computer-readable memory medium of claim15, wherein said providing the videoconference for playback on thedisplay further comprises: determining a second point in thevideoconference when the portion of the second videoconferencinginformation ends using the timeline; and cease providing the secondvideoconference information for playback on the display at the secondpoint.
 17. The non-transitory computer-readable memory medium of claim11, wherein the first videoconference information, the secondvideoconference information, and the timeline are received from anendpoint of the videoconference.
 18. A method for recording avideoconference, comprising: receiving first videoconference informationfor recording during the videoconference, wherein the firstvideoconference information corresponds to participants in thevideoconference, and wherein said receiving the first videoconferenceinformation begins at a first time; receiving second videoconferenceinformation for recording, wherein the second videoconferenceinformation corresponds to a presentation presented during thevideoconference, wherein the second videoconference information isreceived during the reception of the first videoconference information,and wherein said receiving the second videoconference information beginsafter the first time; receiving a timeline of the secondvideoconferencing information for recording during the videoconference,wherein the timeline specifies when the second videoconferencinginformation was present in the videoconference; wherein the timeline isused during playback of the recorded videoconference to indicate whenthe second videoconference information is available for display.
 19. Themethod of claim 18, wherein said receiving the first videoconferenceinformation, said receiving the second videoconference information, andsaid receiving the timeline is performed by a recording server to recordthe videoconference.
 20. The method of claim 18, further comprising:providing the first videoconference information, the secondvideoconference information, and the timeline of the secondvideoconference information to one or more clients, wherein the one ormore clients are configured to use the timeline during playback topresent the second videoconference information.
 21. The method of claim18, wherein said receiving the first videoconference information, saidreceiving the second videoconference information, and said receiving thetimeline is performed by a client to playback the videoconference. 22.The method of claim 18, further comprising: providing thevideoconference for playback on a display, wherein said providingcomprises: providing the first videoconference information for playbackon the display; determining a first point in the videoconference when aportion of the second videoconferencing information begins using thetimeline; and providing the second videoconference information forplayback on the display at the first point.
 23. The method of claim 22,wherein said providing the videoconference for playback on the displayfurther comprises: determining a second point in the videoconferencewhen the portion of the second videoconferencing information ends usingthe timeline; and cease providing the second videoconference informationfor playback on the display at the second point.
 24. The method of claim18, wherein the first videoconference information, the secondvideoconference information, and the timeline are received from anendpoint of the videoconference.
 25. The method of claim 18, wherein thefirst videoconference information is different from videoconferenceinformation provided to videoconference endpoints of thevideoconference.
 26. A non-transitory computer-readable memory mediumcomprising program instructions for streaming a videoconference, whereinthe program instructions execute during a videoconference, wherein theprogram instructions are executable to: provide first videoconferenceinformation to a streaming server for recording during thevideoconference, wherein the first videoconference informationcorresponds to participants in the videoconference, and wherein saidproviding the first videoconference information begins at a first time;provide second videoconference information to the streaming server forrecording during the videoconference, wherein the second videoconferenceinformation corresponds to a presentation presented during thevideoconference, wherein the second videoconference information isprovided during the provision of the first videoconference information,and wherein said providing the second videoconference information beginsafter the first time; provide a timeline of the second videoconferencinginformation to the streaming server for streaming, wherein the timelinespecifies when the second videoconferencing information was present inthe videoconference; wherein the timeline is used while viewing thestreamed videoconference to indicate when the second videoconferenceinformation is available for display.
 27. The non-transitorycomputer-readable memory medium of claim 26, wherein the firstvideoconference information and the second videoconference informationare configured for provision to clients by the streaming server withoutsubstantial modification.